Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis

Knowledge distillation (KD) has proved to be an effective approach for deep neural network compression, which learns a compact (student) by transferring the knowledge from pre-trained, over-parameterized (teacher). In traditional KD, transferred is usually obtained feeding training samples teacher obtain class probabilities. However, original dataset not always available due storage costs or privacy issues. this study, we propose novel data-free KD modeling intermediate feature space of with multivariate normal distribution and leveraging soft targeted labels generated synthesize pseudo as transfer set. Several student networks trained these synthesized sets present competitive performance compared set other approaches.